{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "daf836e3-98fe-f2be-f6c6-ec61d50f3150"
   },
   "source": [
    "In this exploration notebook, we shall try to uncover the basic information about the dataset which will help us build our models / features.\n",
    "\n",
    "Let us start with importing the necessary modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "010c0e14-3312-826e-ecd6-b7db41d41f94"
   },
   "outputs": [],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn import model_selection, preprocessing\n",
    "import xgboost as xgb\n",
    "color = sns.color_palette()\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "pd.options.mode.chained_assignment = None  # default='warn'\n",
    "pd.set_option('display.max_columns', 500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "78b69b00-2ca9-ec50-0e64-df78c0698238"
   },
   "source": [
    "First let us import the train file and get some idea about the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "cf59f829-b0bd-fe3c-cd3b-c9c176c678ea"
   },
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(\"../input/train.csv\")\n",
    "train_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "094c89d2-2c4a-49a4-5536-7d26173f2909"
   },
   "outputs": [],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "4d035267-d1d5-e27c-1f36-887d9bc0359e"
   },
   "source": [
    "There are quite a few variables in this dataset. \n",
    "\n",
    "Let us start with target variable exploration - 'price_doc'. First let us do a scatter plot to see if there are any outliers in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "3b2b42e1-a5e8-79ff-af6b-91b40664e13e"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8,6))\n",
    "plt.scatter(range(train_df.shape[0]), np.sort(train_df.price_doc.values))\n",
    "plt.xlabel('index', fontsize=12)\n",
    "plt.ylabel('price', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "d975201d-d9c3-f456-4dac-4964e964d96b"
   },
   "source": [
    "Looks okay to me. Also since the metric is RMSLE, I think it is okay to have it as such. However if needed, one can truncate the high values. \n",
    "\n",
    "We can now bin the 'price_doc' and plot it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "03e5a2f2-c68b-9965-33dd-b698775c11cf"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "sns.distplot(train_df.price_doc.values, bins=50, kde=True)\n",
    "plt.xlabel('price', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "205d91d5-47ce-f88b-154e-b441c043898b"
   },
   "source": [
    "Certainly a very long right tail. Since our metric is Root Mean Square **Logarithmic** error, let us plot the log of price_doc variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "27b08122-7009-d756-e695-c463a5a27d60"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "sns.distplot(np.log(train_df.price_doc.values), bins=50, kde=True)\n",
    "plt.xlabel('price', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "d80989f8-eae8-bede-f088-4d7c6c8cd7fb"
   },
   "source": [
    "This looks much better than the previous one. \n",
    "\n",
    "Now let us see how the median housing price change with time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "c2cc67da-dd73-7fb7-473f-3987f1a5ecb8"
   },
   "outputs": [],
   "source": [
    "train_df['yearmonth'] = train_df['timestamp'].apply(lambda x: x[:4]+x[5:7])\n",
    "grouped_df = train_df.groupby('yearmonth')['price_doc'].aggregate(np.median).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "18daf0fc-7fc4-f117-adb6-b4d3c2483412"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "sns.barplot(grouped_df.yearmonth.values, grouped_df.price_doc.values, alpha=0.8, color=color[2])\n",
    "plt.ylabel('Median Price', fontsize=12)\n",
    "plt.xlabel('Year Month', fontsize=12)\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "4a23d010-ccde-e93a-56f3-4b1d1774091b"
   },
   "source": [
    "There are some variations in the median price with respect to time. Towards the end, there seems to be some linear increase in the price values.\n",
    "\n",
    "Now let us dive into other variables and see. Let us first start with getting the count of different data types. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "4a510255-4568-c574-0a56-1baeb94e38d1"
   },
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(\"../input/train.csv\", parse_dates=['timestamp'])\n",
    "dtype_df = train_df.dtypes.reset_index()\n",
    "dtype_df.columns = [\"Count\", \"Column Type\"]\n",
    "dtype_df.groupby(\"Column Type\").aggregate('count').reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "636d0880-b0df-cb3c-6a26-633643b03046"
   },
   "source": [
    "So majority of them are numerical variables with 15 factor variables and 1 date variable.\n",
    "\n",
    "Let us explore the number of missing values in each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "2e68a733-1357-4adb-af07-53b7440913c9"
   },
   "outputs": [],
   "source": [
    "missing_df = train_df.isnull().sum(axis=0).reset_index()\n",
    "missing_df.columns = ['column_name', 'missing_count']\n",
    "missing_df = missing_df.ix[missing_df['missing_count']>0]\n",
    "ind = np.arange(missing_df.shape[0])\n",
    "width = 0.9\n",
    "fig, ax = plt.subplots(figsize=(12,18))\n",
    "rects = ax.barh(ind, missing_df.missing_count.values, color='y')\n",
    "ax.set_yticks(ind)\n",
    "ax.set_yticklabels(missing_df.column_name.values, rotation='horizontal')\n",
    "ax.set_xlabel(\"Count of missing values\")\n",
    "ax.set_title(\"Number of missing values in each column\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "0f893fb6-3526-3fd1-9780-db189ca5addf"
   },
   "source": [
    "Seems variables are found to missing as groups.\n",
    "\n",
    "Since there are 292 variables, let us build a basic xgboost model and then explore only the important variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "31677ea2-0321-f709-dac8-120a2d02278f"
   },
   "outputs": [],
   "source": [
    "for f in train_df.columns:\n",
    "    if train_df[f].dtype=='object':\n",
    "        lbl = preprocessing.LabelEncoder()\n",
    "        lbl.fit(list(train_df[f].values)) \n",
    "        train_df[f] = lbl.transform(list(train_df[f].values))\n",
    "        \n",
    "train_y = train_df.price_doc.values\n",
    "train_X = train_df.drop([\"id\", \"timestamp\", \"price_doc\"], axis=1)\n",
    "\n",
    "xgb_params = {\n",
    "    'eta': 0.05,\n",
    "    'max_depth': 8,\n",
    "    'subsample': 0.7,\n",
    "    'colsample_bytree': 0.7,\n",
    "    'objective': 'reg:linear',\n",
    "    'eval_metric': 'rmse',\n",
    "    'silent': 1\n",
    "}\n",
    "dtrain = xgb.DMatrix(train_X, train_y, feature_names=train_X.columns.values)\n",
    "model = xgb.train(dict(xgb_params, silent=0), dtrain, num_boost_round=100)\n",
    "\n",
    "# plot the important features #\n",
    "fig, ax = plt.subplots(figsize=(12,18))\n",
    "xgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "7f10f4aa-05e8-0818-505e-9a5166e6a52b"
   },
   "source": [
    "So the top 5 variables and their description from the data dictionary are:\n",
    "\n",
    " 1. full_sq - total area in square meters, including loggias, balconies and other non-residential areas\n",
    " 2. life_sq - living area in square meters, excluding loggias, balconies and other non-residential areas\n",
    " 3. floor - for apartments, floor of the building\n",
    " 4. max_floor - number of floors in the building\n",
    " 5. build_year - year built\n",
    "\n",
    "Now let us see how these important variables are distributed with respect to target variable.\n",
    "\n",
    "**Total area in square meters:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "407b8469-fcc2-5d41-44de-2b3833523006"
   },
   "outputs": [],
   "source": [
    "ulimit = np.percentile(train_df.price_doc.values, 99.5)\n",
    "llimit = np.percentile(train_df.price_doc.values, 0.5)\n",
    "train_df['price_doc'].ix[train_df['price_doc']>ulimit] = ulimit\n",
    "train_df['price_doc'].ix[train_df['price_doc']<llimit] = llimit\n",
    "\n",
    "col = \"full_sq\"\n",
    "ulimit = np.percentile(train_df[col].values, 99.5)\n",
    "llimit = np.percentile(train_df[col].values, 0.5)\n",
    "train_df[col].ix[train_df[col]>ulimit] = ulimit\n",
    "train_df[col].ix[train_df[col]<llimit] = llimit\n",
    "\n",
    "plt.figure(figsize=(12,12))\n",
    "sns.jointplot(x=np.log1p(train_df.full_sq.values), y=np.log1p(train_df.price_doc.values), size=10)\n",
    "plt.ylabel('Log of Price', fontsize=12)\n",
    "plt.xlabel('Log of Total area in square metre', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "c6ec261b-ba31-57a5-999a-5efcead8dbff"
   },
   "source": [
    "**Living area in square meters:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "e4e720e7-075c-0bed-2982-61c3a97775b0"
   },
   "outputs": [],
   "source": [
    "col = \"life_sq\"\n",
    "train_df[col].fillna(0, inplace=True)\n",
    "ulimit = np.percentile(train_df[col].values, 95)\n",
    "llimit = np.percentile(train_df[col].values, 5)\n",
    "train_df[col].ix[train_df[col]>ulimit] = ulimit\n",
    "train_df[col].ix[train_df[col]<llimit] = llimit\n",
    "\n",
    "plt.figure(figsize=(12,12))\n",
    "sns.jointplot(x=np.log1p(train_df.life_sq.values), y=np.log1p(train_df.price_doc.values), \n",
    "              kind='kde', size=10)\n",
    "plt.ylabel('Log of Price', fontsize=12)\n",
    "plt.xlabel('Log of living area in square metre', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "038f8a9b-0b95-633d-6a8d-0bbfa10116f3"
   },
   "source": [
    "**Floor:**\n",
    "\n",
    "We will see the count plot of floor variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "0c89affd-44b0-6c6c-f616-8bb2c518957a"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "sns.countplot(x=\"floor\", data=train_df)\n",
    "plt.ylabel('Count', fontsize=12)\n",
    "plt.xlabel('floor number', fontsize=12)\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "0e11ee5f-88bb-b092-e522-64c91cee25b0"
   },
   "source": [
    "The distribution is right skewed. There are some good drops in between (5 to 6, 9 to 10, 12 to 13, 17 to 18). Now let us see how the price changes with respect to floors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "a0cae509-6568-0b07-ae4c-f5adf825f709"
   },
   "outputs": [],
   "source": [
    "grouped_df = train_df.groupby('floor')['price_doc'].aggregate(np.median).reset_index()\n",
    "plt.figure(figsize=(12,8))\n",
    "sns.pointplot(grouped_df.floor.values, grouped_df.price_doc.values, alpha=0.8, color=color[2])\n",
    "plt.ylabel('Median Price', fontsize=12)\n",
    "plt.xlabel('Floor number', fontsize=12)\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "af624cc2-1563-02e1-c87b-47b0bfbaacfd"
   },
   "source": [
    "This shows an overall increasing trend (individual houses seems to be costlier as well - check price of 0 floor houses). \n",
    "A sudden increase in the house price is also observed at floor 18.\n",
    "\n",
    "**Max floor:**\n",
    "\n",
    "Total number of floors in the building is one another important variable. So let us plot that one and see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "7bd45c6e-b9e2-c95c-fe31-52e307e2b1ca"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "sns.countplot(x=\"max_floor\", data=train_df)\n",
    "plt.ylabel('Count', fontsize=12)\n",
    "plt.xlabel('Max floor number', fontsize=12)\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "39981f77-9e03-79d5-33c9-91e4325d8ab6"
   },
   "source": [
    "We could see that there are few tall bars in between (at 5,9,12,17 - similar to drop in floors in the previous graph). May be there are some norms / restrictions on the number of maximum floors present(?). \n",
    "\n",
    "Now let us see how the median prices vary with the max floors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "5087aae6-a3e9-fef6-9d05-e8240716a9c1"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "sns.boxplot(x=\"max_floor\", y=\"price_doc\", data=train_df)\n",
    "plt.ylabel('Median Price', fontsize=12)\n",
    "plt.xlabel('Max Floor number', fontsize=12)\n",
    "plt.xticks(rotation='vertical')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "4f21aa45-f21f-50cf-a9a7-0e4e68f333bb"
   },
   "source": [
    "More to come. Stay tuned.!"
   ]
  }
 ],
 "metadata": {
  "_change_revision": 0,
  "_is_fork": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
